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Visual and auditory temporal processing and crossmodal integration are crucial factors in 
the word decoding process. The speed of processing (SOP) gap (Asynchrony) between 
these two modalities, which has been suggested as related to the dyslexia phenomenon, 
is the focus of the current study. Nineteen dyslexic and 17 non-impaired University 
adult readers were given stimuli in a reaction time (RT) procedure where participants 
were asked to identify whether the stimulus type was only visual, only auditory or 
crossmodally integrated. Accuracy, RT and Event Related Potential (ERP) measures were 
obtained for each of the three conditions. An algorithm to measure the contribution 
of the temporal SOP of each modality to the crossmodal integration in each group of 
participants was developed. Results obtained using this model for the analysis of the 
current study data, indicated that in the crossmodal integration condition the presence 
of the auditory modality at the pre-response time frame (between 170 and 240 ms after 
stimulus presentation), increased processing speed in the visual modality among the 
non-impaired readers, but not in the dyslexic group. The differences between the temporal 
SOP of the modalities among the dyslexics and the non-impaired readers give additional 
support to the theory that an asynchrony between the visual and auditory modalities is a 
cause of dyslexia. 
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INTRODUCTION 

According to a widely accepted definition of developmental 
dyslexia, a dyslexic reader is one who exhibits slow and inaccurate 
reading performance unrelated to his/her IQ level or educa- 
tional opportunities (British Psychological Society, 1999; Lyon 
and Shaywitz, 2003; Shaywitz and Shaywitz, 2008). An abun- 
dance of research into phonological deficits has indicated that the 
primary source of the difficulties experienced by dyslexic read- 
ers lies in word decoding accuracy (Liberman and Shankweiler, 
1991; Badian, 1997). Some studies have found that disabled read- 
ers demonstrate a fundamental orthographic deficit (Stanovich 
and West, 1989; Cunningham and Stanovich, 1990; Zecker, 1991; 
Barker et al., 1992; Morris et al., 1998). The reading deficits 
of developmental dyslexia persist into adulthood (Bruck, 1992; 
Leonard et al., 2001). A large number of studies have shown defi- 
cient phonological processing as a core deficit in developmental 
dyslexia. The leading theory, the phonological deficit theory of 
developmental dyslexia (Stanovich, 1988; Share, 1994; Snowling, 
1995), suggests that dyslexic readers may suffer from an (unspec- 
ified) dysfunction in peri-sylvian brain regions, which leads to 
difficulties in generating and processing accurate and efficient 
phonological representations of speech sounds (Stanovich, 1988; 
Temple etal, 2001). 

In addition, researchers are still debating whether dyslexic 
readers are characterized by impairment in basic auditory and/or 
visual processing (Amitay et al, 2002a,b; Vellutino et al., 2004; 



Lachmann et al., 2005; Groth et al., 2011). In the auditory 
domain, data has indicated dyslexics' inability to discriminate 
between temporal rapidly changing tones and consecutive acous- 
tic events (Tallal, 1980; Tallal et al, 1993, 1998). Difficulties 
locating the origin of sounds and blending them were also 
found (Stein, 1993). In the visual domain, findings have indi- 
cated that dyslexic readers have smaller and fewer neurons in 
the magnocellular layers of the Lateral Geniculate Nucleus (LGN) 
(Livingstone, 1991) suggesting fundamental deficiencies of the 
magnocellular system and pathways of the visual cortex that are 
responsible for motion, contrast sensitivity (Stein and Walsh, 
1997; Stein and Talcott, 1999; Stein, 2001; Stein et al., 2001) 
and fast sequential processing (Ben-Yehudah and Ahissar, 2004). 
Based on the aforementioned studies regarding the presumed 
low-level visual and/or auditory sensory deficit among dyslexic 
readers, it was recently demonstrated, using the Pair Associate 
Learning paradigm (Hulme et al., 2007), that dyslexic readers 
exhibited a crossmodal association difficulty (Jones et al, 2013). 
Previous data (Breznitz, 2002, 2003, 2006) has found that the 
gap in the speed of processing (SOP) between the visual and 
auditory modalities is wider among dyslexic readers than non- 
impaired readers. This gap prevents the precise integration over 
time (Berninger et al., 1990; Wolf and Bowers, 2000) of the cross- 
modal information necessary for accurate word decoding and 
leads to the asynchrony phenomenon. Breznitz (2006) suggested 
that the asynchrony phenomenon in the word-decoding process 
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occurs when there is time gap resulting in a mismatch between 
the grapheme and its phoneme correspondent. 

It was previously suggested that different frequency ranges 
are important to speech perception (Luo and Poeppel, 2007; 
Ghitza and Greenberg, 2009). The temporal sampling framework 
for developmental dyslexia (Goswami, 2011) proposes that at 
the basic level of auditory perception processing, dyslexic read- 
ers have difficulties distinguishing between different frequency 
ranges, an impairment that leads to a slower and less accurate 
speech perception (Goswami et al., 2010). It is reasonable to 
assume that an impairment in sensory temporal processing that 
can be identified at the frequency domain may be also expressed 
at the time domain. Thus, it is important to study whether 
dyslexic readers exhibit an abnormal basic sensory (auditory or 
visual) information processing. Here, the dyslexic readers' ability 
to process fundamental sensory input is investigated by using a 
time-based cross-correlation analysis. 

The synchronization hypothesis proposes that for accurate 
information processing to occur, it is crucial that the informa- 
tion that arrives from more than one modality be integrated in 
both content (Fujisaki and Nahida, 2005; Ghajar and Ivry, 2008; 
Neil et al, 2011) and time (Llinas, 1993; Breznitz, 2001, 2002, 
2003, 2006; Breznitz and Misra, 2003; Breznitz et al, 2013). The 
act of reading relies on the information processing system and 
during the word decoding process both the visual and the audi- 
tory modalities are activated. Word decoding accuracy can be 
achieved only when the activation within and between modali- 
ties is synchronized (Breznitz, 2008). It was found that the time 
gap between the visual and auditory Event Related Potential 
(ERP) component of PI of the dyslexic readers was larger than 
100 ms, whereas the non-impaired readers exhibited an insignifi- 
cant time gap of 15-30 ms only (Breznitz, 2008). A similar trend 
of results was found when the time gap between the auditory 
and visual ERP of Nl was analyzed. Moreover, it was shown that 
the between-modalities time gap has developmental constraints: 
while the auditory components of Nl and PI had shorter latency 
as compared to visual ones among dyslexic children, the oppo- 
site was found among adult dyslexic readers (Breznitz, 2008). The 
author argued that these results support the notion that a deficit 
within the visual modality affects decoding ability, a claim that 
was supported by evidence of a significant correlation between the 
between-modalities time gap and reading performance (fluency 
and reading errors). 

The visual-orthographic and auditory-phonological systems 
are the core systems activated temporally during the word- 
decoding process. However, during a normal course of processing, 
both systems differ not only in the structure and length of 
their neural networks, but are also located in different parts of 
the brain (Saito et al., 2005) and operate in a different man- 
ner and at different speeds (Breznitz, 2002, 2003, 2006). Data 
has indicated that auditory information arrives (Mishra et al., 
2007) at the auditory cortex, about 30 ms after stimulus pre- 
sentation (Heil et al, 1999), whereas visual information arrives 
(Mishra et al., 2007) at the visual cortex about 70 ms after stim- 
ulus presentation (Schmolesky et al., 1998). This suggests that 
when a stimulus includes both visual and auditory representa- 
tions, the auditory stimuli arrive in the brain faster than the 



visual stimuli. However, at the word-decoding level, the auditory- 
phonological channel perceives and processes the information 
in a temporal-serial manner (Rosenzweig and Bennet, 1996), 
whereas the visual-orthographic channel process information in a 
spatial, holistic manner (Breznitz, 2006). This suggests that visual 
processing at this level might be faster than auditory processing. 
Yet, an effective word decoding process requires an exact inte- 
gration of graphemes and phonemes (Adams, 1990; Berninger, 
2001). In other words, the two systems need to be synchronized 
for appropriate crossmodal integration to occur. The focus of 
the current study is to quantify the synchronization of the two 
modalities among non-impaired and dyslexic readers. 

Earlier studies that verified the relationships between visual 
and auditory processing among non-impaired readers suggested 
that presenting a visual pattern prior to an auditory one eases 
inter-sensory correspondence while the presentation of an audi- 
tory pattern prior to a visual one increases correspondence errors 
(Botuck and Turkewitz, 1990). It has been suggested that either 
the information might be received more accurately through the 
visual as opposed to the auditory modality, or it is more difficult 
to register information arranged temporally than information 
arranged spatially in memory (Botuck and Turkewitz, 1990). 
Furthermore, Ben-Artzi and Marks (1995) examined whether 
and how stimulus type influences visual-auditory interaction. In 
Ben-Artzi and Marks' study (1995), participants were asked to 
classify sound levels and spatial locations on two types of tasks: 
uni-dimensional and bi-dimensional. Data indicated that visual 
identification is not only faster (see also Melara and O'Brien, 
1987) but also dominant (Egeth and Sager, 1977) over the audi- 
tory identification. Posner et al. (1976) suggested that visual 
dominance is, in fact, a compensation for the visual system's 
limited capacity to arouse internal attention. According to this 
approach, the increases in sound level in the auditory system 
arouse attention automatically. However, the arousal of attention 
in the visual system requires specific controlled effort and the 
brain learns to allocate attention to visual stimuli. Furthermore, it 
has been suggested that the existence of the visual system, which 
can allocate attention to spatial stimuli, enables the ear to relate to 
its relevant stimuli (Posner et al., 1976). By using fMRI imaging 
techniques, recent studies have indicated that during crossmodal 
activation, when both visual and auditory information are pre- 
sented, the visual modality is dominant at the pre- response level 
whereas the auditory is more dominant at the response level (i.e., 
Koppen et al., 2009; Chen and Zhou, 2013). Thus, all the above 
support the notion that a deficit in the visual modality might be 
involved in dyslexia. In light of the assumption that the visual 
modality has dominancy over the auditory modality, it is impor- 
tant to investigate the relationship and interaction between the 
two sensory modalities. 

The basic assumption of this study is that exposing a partici- 
pant to either a visual or an auditory stimulus triggers sequential 
information processing which has an effect on ongoing ERP activ- 
ity. Furthermore, visual-only or auditory-only information pro- 
cessing is carried out in a similar fashion irrespective of whether 
the stimulus is solely visual or accompanied by an auditory 
stimulus presented at the same time (crossmodal presentation). 
Therefore, the first hypothesis of this study is that the ongoing 
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ERP of the crossmodal condition will contain visual and auditory 
elements that will also be found in each of the unimodal con- 
ditions (Marchant and Driver, 2013). It should be noted that 
the current study does not assume that an apparent correlation 
between the unimodal and crossmodal ERPs stems solely from the 
presence of the unimodal element within the crossmodal ERP. It 
is proposed that to a certain extent, the two datasets share a com- 
mon factor which may be explained as a specific unimodal brain 
reaction. 

The second hypothesis asserts that although the information 
processing of a unimodality (specific-visual or auditory) will be 
similar under the two conditions of the unimodal stimulus type 
and under the two modalities stimulus type, the crossmodal pro- 
cessing will affect the SOP of the uni (specific) modality. Thus, 
if the data were to be looked at in an individual time window for 
the unimodal condition, it is assumed that a similar pattern of the 
component (factor) will be found in a corresponding time win- 
dow for the crossmodal condition, but not necessarily in exactly 
the same time location, as it may appear earlier or later [this is 
defined as Delta Time (DT)]. The polarity of DT indicates either 
an acceleration (negative DT) or deceleration (positive DT) in 
one modality's SOP as a result of the presence of the second 
modality. 

As the SOP of the two different modalities was shown to dif- 
fer (Saito et al, 2005) the third hypothesis of the current study is 
that differences will be found between the size of the interaction 
between the visual and auditory modalities and the influence of 
one modality on the other. Moreover, as dyslexic readers' ability to 
process uni (visual or auditory) and crossmodal sensory informa- 
tion was found deficient (Lachmann et al., 2005; Breznitz, 2006; 
Jones et al., 2013), the fourth hypothesis of the current study 
is that differences between the two-reading-level groups will be 
found. Specifically, it is assumed that the results of the current 
study would show a negative effect of one modality (visual or 
auditory) on the SOP of the other modality among the dyslexic 
readers. 

METHODS 
PARTICIPANTS 

Nineteen dyslexic readers (age 25.5 ± 2.91) and 17 non-impaired 
readers (age: 24.52 ± 2.29) were included in the study [£(35) = 
0.191,p = 0.242]. None of the participants had a history of neu- 
rological or emotional disorders, and no differences were found 
between dyslexics and non- impaired readers in attention ability 
as measured by the d2 test for adults [£(35) = —0.222, p = 0.825] 
(Brickenkamp, 1981). The dyslexic readers were diagnosed with 
dyslexia during childhood, and their diagnosis was confirmed 
as adults by achieving one standard (—1) score or below on 
the Hebrew "MATAL" normative reading test (MATAL Battery, 
2007). The non-impaired readers were recruited via notices 
posted on campus bulletin boards. Individuals with dyslexia were 
recruited through the Student Support Service at the University 
of Haifa. All participants were native Hebrew speakers from a 
middle-class background, right-handed, and screened for nor- 
mal hearing. All participants displayed normal or corrected-to- 
normal vision in both eyes. All participants gave their informed 
written consent prior to inclusion in the study, and all were paid as 



compensation for their time. The experiment was approved by the 
University of Haifa Ethics Committee (Number, 1991) according 
to the Helsinki Declaration. 

The classification of participants into non-impaired readers 
and dyslexic readers groups was based on the following behavioral 
measures (For more details see Breznitz and Misra, 2003; Breznitz 
etal, 2013). 

Intelligence 

Intelligence was tested by the Block Design (performance) and 
the Similarities sub-tests (verbal) from the WAIS-III (Wechsler, 
1997). 

Decoding skills 

One Minute Tests (Breznitz and Misra, 2003) comprised a battery 
of two subtests one for words and the other for pseudowords in 
which the participants were asked to read single words or pseu- 
dowords as quickly and as accurately as possible within the space 
of 1 min. 

Reading rate and accuracy of connected text 

Oral reading time and accuracy of a narrative text comprising 247 
words (MATAL Battery, 2007. See also Breznitz and Misra, 2003). 
Reading time was defined as the mean reading time for each word 
read correctly. 

Reading rate and comprehension 

A Reading Comprehension Test (MATAL Battery, 2007), com- 
prising 412 words. The participants were asked to silently read a 
passage as quickly as possible and then answer 18 comprehension 
questions. Reading time was based on the mean reading time per 
word. Comprehension scores were based on the total number of 
correct answers. 

Memory 

In the Digit Span (WAIS III, Wechsler, 1997), the standard scores 
of each participant were recorded. 

Speed of processing (SOP) 

Two tests were used to assess SOP: The Digit Symbol Task and 
Coding Task- Speed Factor (WAIS III, Wechsler, 1997). 

Table 1 presents the means, standard deviations, and t val- 
ues for the reading and the cognitive background measures. The 
dyslexic readers achieved significantly lower scores than the non- 
impaired readers in reading accuracy and time parameters but 
not for silent reading comprehension. In addition the dyslexic 
readers also obtained significantly lower scores compared to the 
non-impaired readers in the SOP and working memory measures 
but not in the intelligence measures (see Table 1). 

APPARATUS 

Two computer sets were used in this study. The first computer 
was used to present the task stimuli (visual as well as auditory, 
using a screen, and a pair of speakers) and to record participant's 
responses. The electrophysiological data was collected using a 
Bio-Logic Brain Atlas IV computer system (2nd computer set) 
with 20 electroencephalographic (EEC) activity reception chan- 
nels. The data collection from the scalp began at approximately 
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Table 1 | Reading and cognitive measures for dyslexic and 
non-impaired readers. 
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2.40 
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4.70 


1.06 


(correct answers out of 18 
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Digit symbol SD score 


12.60 


10.75 


8.23 


15.16 


-2.63** 


Coding SD Score 


13.87 


10.86 


8.90 


5.98 


5.57*** 


General ability similarities 


12.01 


3.86 


11.90 


2.91 


1.57 


Block design 


12.67 


2.29 


12.85 


1.66 


1.21 


Digit span — standard score 


12.42 


3.08 


9.00 


2.76 


-4.38*** 



**p<0.01; ***p< 0.001. 



1000 ms prior to the beginning of the experimental task. The 
sample rate was 256 Hz and was carried out using a full array 
of electrodes placed according to the International 10/20 sys- 
tem (Jasper, 1958) utilizing an Electro-cap (a nylon cap fitted 
over the head with 9 mm tin electrodes sewn within). An electro- 
oculogram (EOG) was recorded with an electrode extension that 
was located under the left eye. A ground electrode was placed on 
the left mastoid. All electrodes were maintained at an impedance 
of 5 KQ, or less. Brain activity was sampled directly from 19 scalp 
electrodes (Fpl, Fp2, F7, F3, Fz, F4, F8, T3, C3, Cz, C4, T4, T5, 
P3, Pz, P4, T6, Ol, 02). 

TASK DESIGN 

A visual, auditory, and crossmodal processing task (Breznitz and 
Misra, 2003; Meyler and Breznitz, 2003) was administered. This 
task consisted of 150 stimuli presented to the participants in 
three different conditions: auditory alone (50 tones occurring at 
1000 Hz with a time length of 200 ms), visual alone (50 white rect- 
angle shape stimuli, presented at the middle of a black screen 
for duration of 200 ms), and crossmodal (50 tones and flashes 
occurring simultaneously). The 150 stimuli were presented in a 
randomized order. The between trials interval, i.e., the time from 
the beginning of one trial to the beginning of the next trial was set 
to 2 s. The participants were asked to press one of three computer 
keys — One key to indicate the appearance of an auditory tone 
stimulus alone, one key for the visual rectangle-like flash, and the 
another key to indicate when the two stimuli occurred simultane- 
ously. Off-line analysis differentiated between the auditory, visual, 
and simultaneous segments. All stimuli were presented to the par- 
ticipants on a PC computer. Participants were seated 0.5 m from 
the computer screen and heard the tones via speakers. 



PROCEDURE 

During data collection, participants were seated in a sound atten- 
uated room. The experiment took place during two sessions of 
about 2 h each. The first part of data collection consisted of 
gathering the behavioral measures, and in the second part the 
experimental tasks ERP measures were incorporated. 

DATA ANALYSIS 

For each participant and for each of the three task conditions, 
both the mean value of Reaction Time (RT) for correct responses 
and accuracy were computed. The EEC data were segmented 
into 1945 ms epochs (one per trial) and by three different con- 
ditions based on the trial type (1 -visual-only, 2-auditory-only, 
3-crossmodal stimulus). Data of one trial started 445 ms prior 
to stimulus presentation and ended 650 ms following presenta- 
tion. The data was corrected for eye movements using the Orgil 
Medical Equipment (1997), normalized by comparing cognitive 
activity time with inactive time for each electrode, separately, 
for each participant, filtered using a low pass 20-24 Hz filter, 
and averaged by stimulus type (Orgil software, 1997) prior to 
beginning the cross-correlation analysis. 

CROSS-CORRELATION ANALYSIS 

In order to verify the research hypotheses, a cross-correlation 
analysis was obtained. Cross-correlation analysis is a proce- 
dure used in signal processing by which the similarity between 
two signals is measured. The algorithm analysis output is a 
series of correlation coefficients between two signals accord- 
ing to time, i.e., one correlation coefficient is calculated for 
successive time points according to a specific temporal incre- 
ment (Woody, 1967; Nelson-Wong et al., 2009). Previously, the 
cross-correlation technique was used as part of a tool aimed 
at overcoming the limitations of the traditional ERP averag- 
ing method and to categorize participants based on their ERP 
data (Sela et al., 2008). Similar to the work of Sela et al. 
(2008), the current study algorithm obtained and compared 
data time windows taken from participants' ERPs. These data 
time windows included all 19 electrodes within the selected 
time areas. In the following algorithm, the term "unimodal" 
refers to either visual- or auditory-only ERP datasets. The 
term "crossmodal" refers to the synchronized visual-auditory 
stimulus's ERP. 

TIME WINDOW ANALYSIS— CREATION OF A CROSS-CORRELATION 
GRAPH 

The following algorithm was used on three different datasets: 
visual-only ERP, auditory-only ERP, and crossmodal ERP. The 
explanation below describes the process in which a cross- 
correlation graph is computed. 

1. A unimodal time window was constructed by taking the val- 
ues (amplitude) of all electrodes from the unimodal dataset 
across a certain time window (Figure 1). Thus, a time win- 
dow is defined as a two-dimensional dataset with electrodes 
as rows and time samples as columns. The time window's 
location is defined as the center of the specific time window. 
For example, a time window taken at the time area of 150 ms 
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Unimodal temporal time window 




trial 



FIGURE 1 | Cross-correlation analysis between a unimodal time 
window throughout the crossmodaltrial. 



is defined as the amplitude data of all electrodes between 
100 and 200 ms (see Temporal implementation of Phases 
A and B for an explanation of the choice of time window 
location). 

2. A cross-correlation loop between the above time window and 
the crossmodal ERP was applied as follows: 

a. A crossmodal ERP time window of the same duration of 
the unimodal ERP time window constructed in Phase 1 was 
computed. The first crossmodal ERP time window used in 
the algorithm was from 0 to 100 ms. Note that the algo- 
rithm takes into account a time window of all electrodes 
altogether. 

b. The correlation strength between the two time windows 
was calculated and resulted in a number within the range 
of — 1 and 1 . A value of zero indicates no correlation. A cor- 
relation value that approaches —1 indicates a strong nega- 
tive correlation and a value that approached 1 indicates a 
strong positive correlation. 

c. A new crossmodal ERP time window was then constructed 
which was located one successive time frame (3.9 ms) after 
the crossmodal ERP time window constructed in Phase 2a. 

d. This procedure was repeated until the end of the cross- 
modal ERP trial duration (see Figure 1). 

3. The result of the algorithm is a series of correlation coefficients 
computed for each unimodal ERP time window location every 
10 ms, from the beginning of the ERP trial to the end. 

If assumption 1 is correct, then the cross-correlation graph 
should appear as in Figure 2. 

Thus, for most of the iterations (Phases 2b to 2d) the correla- 
tion between the unimodal ERP window and the crossmodal 
ERP should be relatively low, but at a particular time, the cor- 
relation strength should increase and reach a relatively high 
peak. The peak's location is the point in time at which the ERP 
data of the unimodal window have their maximum similarity 
to the crossmodal ERP. This notion can be understood as two 
pictures that look almost the same. 

4. The DT between the peak time and the center of the unimodal 
window was measured. DT is the parameter used as the basis 
for the next phase of analysis. 




Time (ms) 



FIGURE 2 | Cross-correlation strength graph. Note the peak in graph 
strength in the area of the unimodal time window (right red dashed 
line in B) and the delta time (DT) between the line and the peak 
maximum. 



DELTA TIME (DT) PARAMETER 

The dependent variable in the proposed algorithm consists of the 
DT between the unimodal window's location and the correlation 
peak's time location. If the unimodal time window represents a 
temporal fraction of information processing and there is a rel- 
atively strong correlation between the window and a window 
similar in size from the crossmodal ERP, then the algorithm sug- 
gests that the same temporal fraction of information processing 
occurred in both ERPs. In addition, if there is a DT between 
the peak location and the unimodal window, then the fraction 
of information processing occurred earlier or later, depend- 
ing upon the DT polarity. Thus, a negative DT indicates early 
occurrence of the information processing fraction, or in other 
words, the second modality influenced and accelerated the pro- 
cess of the first modality. In contrast, a positive DT indicates 
late occurrence of the information processing fraction, which 
means that the second modality decelerated the SOP of the first 
modality. 

The algorithm found the DT for each participant between the 
peak's location and the center of the unimodal time window. A 
f-test assessed whether there was a significant difference on DT 
values between the two groups of participants. 

TEMPORAL IMPLEMENTATION OF PHASES A AND B 

The phases described above (A and B) focus on a particular 
time window (for example, between 100 and 200 ms). However, 
SOP rate can change throughout time. In addition, it is reason- 
able to assume that the second modality's degree of influence 
is varied at different time locations. Therefore, the last phase 
of the algorithm runs across the entire unimodal time course. 
In other words, the DT for each time window's location for 
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each participant was calculated between the location of the uni- 
modal window and the location of the correlation peak with 
the crossmodal ERR For each of the time windows, an aver- 
age DT was computed for each group. It has been suggested 
(Goswami, 2011) that dyslexic individuals suffer from difficul- 
ties at processing information presented at rates corresponding 
to the occurrence of syllabic information in speech, and crit- 
ical for speech comprehension (4-7 Hz, Giraud and Poeppel, 
2012). Information at these frequencies arrive to the brain every 
142-250 ms. Therefore, it is reasonable to assume that in the 
current study, where the task was to react to a single event 
stimulus (visual, auditory, or crossmodal type), the DT of each 
of the groups would differ statistically within this time area. 
To verify this assumption, a series of f-test analyses was run 
on the DT parameter (that was computed based on each of 
the time windows taken from the time area of 140-250 ms) to 



assess whether there was a significant difference between the two 
groups. 

IMPLEMENTATION OF PHASE C ON BOTH MODALITIES 

The procedure was run twice, once for each modality. The win- 
dow locations were set between 50 and 650 ms. Each successive 
time window was moved in increments of 10 ms (i.e., the sec- 
ond time span was centered at 60 ms, the third at 70 ms, etc.; see 
Figure 3 for a full flowchart description of the cross-correlation 
algorithm). 

RESULTS 

EXPERIMENTAL BEHAVIORAL MEASURES: 

In order to investigate task accuracy and RT when process- 
ing visual, auditory and crossmodal integration, an analysis 
of variance (ANOVA) in a 2x3 design (Group (dyslexic 



A, B 



Construct time window from unimodal 

EKP 



Correlate unimodal time window with a 
size matched window from the 
synchronized ERP 



CCA 



Construct successive time window from 
the synchronized ERP 



Create Cross Correlation graph 



Locate peak. If peak is notable then 
find DT. Register DT for later analysis. 



Take successive time window from 
unimodal ERP 



Run the algorithm twice: once on the 
visual modality, and again on the 
auditory modality. 



FIGURE 3 | The cross-correlation algorithm flow chart. 
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Xnon-impaired readers) x conditions (visual-only x auditory- 
only x crossmodal) was employed for RT and Accuracy 
separately. No significant between-group differences were 
found in accuracy [i 7 ^ 34 ) = 2.48, p > 0.05]. A significant 
Condition effect was obtained [F(2, 68) = 50.2, p < 0.001] 
which stemmed from a lower performance under the audi- 
tory only condition for both groups (Table 2). No significant 
group by Condition interaction was found [_F( 2 , 68) = 0.22, 
p > 0.05]. 

The analysis of the RT data revealed a significant group effect 
[_F(! 34) = 4.45, p < 0.05], but no significant condition effect 
[F(2, 68) = 1-2, p > 0.05] and no significant group by condition 
interaction [P(2, 68) = 1-44, p > 0.05]. 

ERP CROSS-CORRELATION ANALYSIS RESULTS 

In order to investigate the influence of the presence of one modal- 
ity on the SOP of the other modality, several cross-correlation 
analyses were obtained on averaged on-going ERR The first anal- 
ysis was obtained in order to investigate the influence of the 
auditory modality on the SOP of the visual one and the second 
analysis investigated the influence the visual modality on SOP of 
the auditory one. The cross-correlation analysis was run between 
sequential time windows taken from each unimodal condition 
and the crossmodal condition. The cross-correlation analysis 
outcome measure, the visual and auditory DT, was then used 
to compare between-modalities' influence and between group 
differences. 



Table 2 | Mean (and Standard Deviation) for the dyslexic and 
non-impaired readers of the behavioral reading and experimental 
measures. 





Dyslexic 


Non-impaired 


r 


P 




readers 


readers 






Visual correct responses (%) 


73 (20.8) 


83 (16) 


1.43 


n.s 


Auditory correct responses (%) 


65 (19.5) 


73 (16.3) 


1.02 


n.s 


Visual + Auditory correct 


71.8 (18.7) 


83.3 (14.2) 


1.68 


n.s 


responses (%) 










Visual reaction time (ms) 


669 (115) 


565 (122) 


2.23 


<0.05 


Auditory reaction time (ms) 


692 (110) 


586 (129) 


2.29 


<0.05 


Visual + auditory reaction time 


639 (136) 


587 (66) 


1.12 


n.s 



(ms) 



Based on the size of the correlation between the unimodal 
data and the crossmodal ERP data within an allotted time 
frame, it can be inferred that a given time window from the 
unimodal condition exists in the crossmodal ERP data (see 
Figure 4). 

For most participants, a notable peak could be identified (see 
Figure 4A), though results should be interpreted with caution as 
the graph's shape did not always indicate successful correlation 
strength (see Figures 4B,C). For example, iteration B's dataset is 
too noisy to determine the peak's location because of an artifi- 
cial waveform and iteration C's is very local and low. Therefore, 
inclusion conditions were developed and applied: only cross- 
correlation graphs which had a peak which was located no more 
than 50 ms before or after the time of the unimodal window 
were included in the computation of the DT average. The time 
window total width of 100 ms (50 ms before and after the time 
window location) was based on previous evidence that asserted 
that the variance of ERP component time locations is normally 
distributed in a time area of less than 50 ms before and after the 
component mean time location (for example, see Simon et al., 
2007; Maurer et al, 2008; Spironelli and Angrilli, 2009). In addi- 
tion, the graphs could not contain more than 3 additional local 
peaks within the given time window. These criteria kept dis- 
torted datasets out of the further analysis (see Phase B). Figure 5 
shows the percentage of participants included in the process in 
every time window location. Figures 5A,B reveal the percentage 
of participants included in each time window analysis throughout 
the visual and auditory information process analysis, respectively. 
Overall, a relatively low number of participants from both groups 
were included in the visual analysis in the time areas of 50-150 
and 450 ms to the end (Figure 5A). It is interesting to note that 
toward the end of the trial, the percentage of dyslexic readers 
included in the analysis decreased more slowly than the percent- 
age of included non-impaired readers. In the auditory analysis 
(Figure 5B), the percentage rate of included participants in both 
groups remained constant from the beginning of the trial until 
about 500 ms. 

In order to investigate the influence of one modality's acti- 
vation on the SOP of the other modality, a DT for each of 
the participants and for each of the time windows was com- 
puted (Figure 6). A f-test analysis was applied for time windows 
located within the time area of 140-250 ms (Table 3), to deter- 
mine if there was a between-group difference in each of the 



CO 

c 

'L' 



o 
|_| 



Iteration A 




Iteration B 




Iteration C 



Time (ms) 



Time (ms) 



FIGURE 4 | Examples of a successful correlation result (Iteration A) 
and unsuccessful correlation result (Iterations B and C). Iteration (A) 
produced a positive peak near the original time of the unimodal 



window. Iteration (B)'s dataset is too noisy to suppose that the value 
of DT is reliable. Iteration (C)'s peak is to low and expresses a weak 
correlation between the two datasets. 
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A Percentage of participants that were included in the 
visual process analysis 




o i 1 1 i 1 1 1 1 

0 100 200 300 400 500 600 700 

Time(ms) 

B Percentage of participants that were included in the 
auditory process analysis 




o -I , , , , , , 1 

0 100 200 300 400 500 600 700 

Time (ms) 

— ♦— Dyslexic readers —■—Skilled readers 

FIGURE 5 | The percentage of participants included in every 
comparison. Note the inverted 'U' shape of the visual process graph where 
at the beginning and end of the process the percentage of participants is 
relatively low compared to the middle. In contrast to the visual process's 
graph, the auditory graph is stable at the beginning. 



time windows (it is important to note here that a f-test was 
used to assess between-group differences rather than repeated 
measures analysis of variance (rmANOVA) due to the num- 
ber of degrees of freedom and the relatively insufficient num- 
ber of participants in this study). When the DT was based 
on the influence of the auditory modality on the SOP of 
the visual modality, significant between-group differences were 
found in the time area of 170 ms through 240 ms (Figure 6A, 
Table 3). In contrast, no significant between-group differences 
were found when the DT was based on the influence of the 
visual modality on the SOP of the auditory modality (Figure 6B). 
Nevertheless, note the positive peak (de-acceleration) in the value 
of the non-impaired readers' DT in the time area of 240 ms, 
and of the dyslexic readers' similarly shaped peak occurring 
at 210 ms. 

DISCUSSION 

The purpose of this experiment was to investigate the effect of 
one modality on the SOP of the second modality. Stimuli of 
visual and auditory modalities were employed in a unimodal 
and crossmodal presentation. Overall, our results support the 
notion that there is in fact an interaction between modalities 
during information processing, consistent with previous research. 
Furthermore, the significant differences that were found between 
dyslexic and non-impaired readers highlight the difficulty which 



A The influence of the presence of the auditory modality 
on the SOP of the visual modality 



60 -i 

t 




Time (ms) — ♦— Dyslexic readers -•— Skilled readers 



B The influence of the presence of the visual modality on 
the SOP of the auditory modality 

60 -, 

t 




Time(ms) 

FIGURE 6 | Average DT of the two groups for each time window. 

Axis X represents time and axis Y represents DT. (A) The DT results 
of the visual-only and the crossmodal ERPs. (B) The DT results of 
the auditory-only and the crossmodal ERPs. 



Table 3 | The comparison between the groups' DT computed from the 
Visual modality time windows taken from the time area of 
140-250 ms. 



Window 
location (ms) 


f (35] 


P 


140 


1.157 


0.258 


150 


1.595 


0.122 


160 


1.426 


0.167 


170 


2.506 


0.019 


180 


2.553 


0.016 


190 


2.872 


0.007 


200 


3.033 


0.005 


210 


3.645 


0.001 


220 


3.022 


0.005 


230 


2.7 


0.011 


240 


2.199 


0.036 


250 


1.022 


0.315 



dyslexic readers face in synchronizing processing between the 
visual and auditory modalities. 

Hypothesis 1 suggested that the ERP data of the crossmodal 
condition would contain unimodal visual and auditory elements. 
It is suggested that the peak in the correlation graph results 
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supports this assumption (Figure 2A). As described previously, a 
cross-correlation analysis computing successive correlation coef- 
ficients between unimodal ERP within a specific time window 
and successive crossmodal ERP time windows sliding along the 
entire crossmodal trial was implemented. More specifically, the 
current algorithm took a fraction (i.e., 100 ms) of the temporal 
information processing from a unimodal ERP trial and searched 
for a matching fraction of information processing throughout 
the crossmodal ERP trial. It is apparent from Figure 2A that 
throughout the sequence, the correlation strength was mainly 
low and randomized. However, at a particular point in time, an 
increase in the correlation strength appeared, usually a short dis- 
tance in time from the time location of the unimodal window. 
It may be argued that this constitutes evidence for the presence 
of the same fraction of information processing that occurs in 
the crossmodal ERP. A short while after the correlation strength 
reaches its peak, it starts to decline as the unimodal window 
passes its identical fraction of information processing and contin- 
ues toward the time area in which low and randomized similarity 
appear. 

Hypothesis 2 implied that the presence of a second modal- 
ity will have an effect on the SOP of a first modality. Figure 2B 
reveals that the correlation graph's peak location is not in the 
exact location of the unimodal time window. Furthermore, data 
analysis across the entire time domain and across participants 
revealed that the correlation peak location may appear shortly 
before or after the unimodal window's time location. It is sug- 
gested that the time difference between the two locations (i.e., 
DT) is a reflection of the degree of influence of the second 
modality on the first modality's SOP when presented simulta- 
neously. If the closest matching fraction of information pro- 
cessing taken from a unimodal ERP was found at a later time 
location (i.e., DT is positive), then this comparison indicates 
that there was a delay in the processing of the modality in 
the crossmodal ERP, meaning that SOP decelerated. Conversely, 
if the DT was negative, then the location of the same frac- 
tion of information processing existed at an earlier point of 
time in the crossmodal ERP compared to the unimodal ERP. 
Consequently, this constitutes evidence of SOP acceleration in the 
modality. 

The neurobiology of the visual system is considered to be 
constructed from two major pathways (the dorsal and ventral), 
where visual information enters the retina and travels to the visual 
cortex. Information reaches the visual cortex about 100-150 ms 
post visual stimulus presentation. Auditory processing begins at 
the ear and initial auditory information reaches the auditory 
cortex through several pathways about 70-100 ms post stimu- 
lus presentation (see Breznitz, 2006 for review). Both modality 
pathways prepare initial sensory information for higher cogni- 
tive processing. As the pathways process information rapidly and 
in areas deep within the brain, it is expected that their oper- 
ations will not have a strong impact on EEC data. Therefore, 
comparing the visual-only ERP and the crossmodal ERP dur- 
ing the first 150 ms of data collection will result in a relatively 
lower "success" rate when locating the visual information pro- 
cessing fraction in the crossmodal ERP (Figure 5A). However, as 
time progresses and processing moves from deeper brain areas 



to higher cortical structures, the size of the common compo- 
nents in the two ERP sets increases. Therefore, the algorithm has 
more success in finding common visual components 150 ms post 
stimulus presentation. As a result, a higher percentage of par- 
ticipants are included in the analysis from 150-450 ms. When 
finding the common components between the auditory-only ERP 
and crossmodal ERP, a higher "success" rate was found at an 
earlier time frame (50 ms post stimulus presentation) as ini- 
tial auditory processing is faster than initial visual processing 
(Figure 5B). 

Shifting our focus to the end of processing within the visual 
modality, a differential between the dyslexic reading group and 
non-impaired readers is apparent beginning at around 450 ms 
post stimulus presentation (Figure 5A). The differential begins 
at this point in time due to the decrease in the percentage of 
non-impaired readers included in the analysis. The percentage of 
dyslexic readers included in the analysis remains high for a longer 
period of time. The decrease in the percentage rate of included 
participants for analysis implies that among the non-impaired 
readers, the shared elements of the two ERPs end at that point. 
Thus, the duration of visual information processing is approxi- 
mately 450 ms, and, as such, the ERP data appearing after 450 ms 
no longer deals with visual information, which results in a lower 
"success" rate of finding a sufficient correlation between the two 
datasets. However, visual information processing lasts longer in 
the dyslexic group and as such, the percentage of participants 
starts to decrease approximately 50 ms later. This is an impor- 
tant difference between the two groups as it provides evidence 
for slow speed of visual processing among the dyslexic readers as 
compared to non-impaired readers. 

As can be seen in Figure 6A, non-impaired readers' visual pro- 
cessing "benefits" from the presence of the auditory synchronized 
stimulus as the value of DT is negative almost throughout the 
entire timeline (i.e., their SOP of visual information is acceler- 
ated). A similar but lower effect was observed among the dyslexic 
readers. Furthermore, a significant difference was found between 
the values of DT appearing around 170 and 240 ms where the DT 
of the non-impaired readers was more negative than that of the 
dyslexic readers. This time area is considered to be related to per- 
ception and has been suggested by previous research to be related 
to the dyslexia phenomenon (Maurer et al., 2006, 2008). On the 
other hand, processing within the auditory modality was not 
affected by the presence of a visual stimulus appearing from about 
50-200 ms in both groups (Figure 6B). The non-impaired read- 
ers began to decelerate their auditory SOP around 210 ms. Similar 
deceleration was observed among the dyslexic readers but 40 ms 
earlier. Based on the reasoning that visual processing is slower 
than auditory processing, we provide additional evidence that in 
the normal information processing sequence, whenever there is 
a need to synchronize the two modalities, the brain accelerates 
processing within the visual modality and decelerates processing 
within the auditory modality. Unlike the non-impaired readers, 
the dyslexic readers do not accelerate their visual SOP to the same 
degree (Figure 6A). Moreover, they decelerated their auditory 
processing too early (Figure 6B). It is possible that this leads to 
asynchrony within the two modalities and to an overall slowness 
of information processing. 
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Failure at the level of lower sensory processing, which was the 
focus of the present study, may accumulate in the higher order 
levels of processing such as processing letters, syllables, words, 
sentences, and general reading comprehension. As reading acti- 
vates sequential parallel visual and auditory processes, synchro- 
nization between the two modalities is necessary for successful 
reading accuracy and rate. The results obtained in the current 
study concerning common elements between single modality 
and crossmodal processing lend support to the synchronization 
hypothesis (Breznitz and Misra, 2003). 

Prior studies in the field of temporal processing have been 
focused on the ability of the brain to process input at different 
frequencies (Buzsaki and Draghun, 2004; Luo and Poeppel, 2007; 
Power et al., 2012). Specifically, the temporal sampling framework 
of dyslexia (Goswami, 2011) suggests that the dyslexic reader may 
suffer from atypical processing of information occurring at fre- 
quencies between 4 and 7 Hz, i.e., every 142-250 ms (theta band, 
and possibly lower frequencies). The current results indicate an 
apparent failure of dyslexic readers in processing information 
occurring within the specific time area of 150-250 ms follow- 
ing a single event stimulus (Figure 6). The non-impaired readers 
demonstrated a non-symmetric effect of one modality on the 
other, in which the occurrence of bimodal information process- 
ing accelerated the SOP of the visual modality and decelerated the 
SOP of the auditory modality at the time area of 170-250 ms. This 
concurrent change in the two modalities' SOP was not obtained 
among the dyslexic readers as their visual modality's SOP did 
not accelerate in the presence of auditory information processing 
(Figure 6). It could be that, as proposed by the temporal sam- 
pling framework of dyslexia (Goswami, 2011), atypical neural 
oscillations at the theta frequency band for auditory processing 
among the dyslexic readers impact negatively the SOP of their 
visual modality processing, by preventing it from accelerating 
adequately. Thus, it may be suggested that these results provide 
additional support for the difficulty of the dyslexic individual to 
process a 4-7 Hz inflow of information and moreover, support 
the temporal sampling framework of dyslexia (Goswami, 2011; 
Power etal., 2012). 
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